U.S.  DEPARTMENT  OF  COMMERCE 
National  Technical  Information  Service 

AD-A026  193 


BIOCYBERNETIC  CONTROL  IN  MAN-MACHINE  INTERACTION 


California  University 


PREPARED  FOR 

Office  of  Naval  Research 


March  1976 


r 


KEEP  UP  TO  DATE 


Between  the  time  you  ordered  this  report— 
which  is  only  one  of  the  hundreds  of  thou- 
sands in  the  NTIS  information  collection  avail- 
able to  you — and  the  time  you  are  reading 
this  message,  several  new  reports  relevant  to 
your  interests  probably  have  entered  the  col- 
lection. 

Subscribe  to  the  Weekly  Government 
Abstracts  series  that  will  bring  you  sum- 
maries of  new  reports  as  soon  as  they  are 
received  by  NTIS  from  the  originators  of  the 
research.  The  WGA’s  are  an  NTiS  weekly 
newsletter  service  covering  the  most  recent 
research  findings  In  25  areas  of  Industrial, 
technological,  and  sociological  Interest — 
invaluable  Information  for  executives  and 
professionals  who  must  keep  up  to  date. 

The  executive  and  professional  informa- 
tion service  provided  by  NTIS  In  the  Weekly 
Government  Abstracts  newsletters  will  give 
you  thorough  and  comprehensive  coverage 
of  government-conducted  or  sponsored  re- 


search activities.  And  you’ll  get  this  Impor- 
tant Information  within  two  weeks  of  the  time 
It’s  released  by  originating  agencies. 

WGA  newsletters  are  computer  produced 
and  electronically  photocomposed  to  siash 
the  time  gap  between  the  release  of  a report 
and  its  availability.  You  can  learn  about 
technical  Innovations  immediately — and  use 
them  in  the  most  meaningful  and  productive 
ways  possible  for  your  organization.  Please 
request  NTIS-PR-205/PCW  for  more  Infor- 
mation. 

The  weekly  newsletter  series  will  keep  you 
current.  But  learn  what  you  have  missed  In 
the  past  by  ordering  a computer  NTISearch 
of  all  the  research  reports  in  your  area  of 
Interest,  dating  as  far  back  as  1964,  if  you 
wish.  Piease  request  NTIS-PR-186/PCN  for 
more  information. 

WRITE:  Managing  Editor 

5285  Port  Royal  Road 
Springfield,  VA  22161 


Keep  Up  To  Date  With  SRIM 


SRIM  (Selected  Research  in  Microfiche) 
provides  you  with  regular,  automatic  distri- 
bution of  the  complete  texts  of  NTIS  research 
reports  only  in  the  subject  areas  you  seiect. 
SRIM  covers  almost  all  Government  re- 
search reports  by  subject  area  and/or  the 
originating  Federal  or  local  government 
agency.  You  may  subscribe  by  any  category 
or  subcategory  of  our  WGA  (Weekly  Govern- 
ment Abstracts)  or  Government  Reports 
Announcements  and  Index  categories,  or  to 
the  reports  issued  by  a particular  agency 
such  as  the  Department  of  Defense,  Federal 
Energy  Administration,  or  Environmental 
Protection  Agency.  Other  options  that  will 
give  you  greater  selectivity  are  available  on 
request. 

The  cost  of  SRIM  service  is  only  45f* 
domestic  (60£  foreign)  for  each  complete 


microflched  report.  Your  SRIM  service  begins 
as  soon  as  your  order  Is  received  and  proc- 
essed and  you  will  receive  biweekly  ship- 
ments thereafter.  If  you  wish,  your  service 
will  be  backdated  to  furnish  you  microfiche 
of  reports  issued  earlier. 

Because  of  contractual  arrangements  with 
several  Special  Technology  Groups,  not  all 
NTIS  reports  are  distributed  In  the  SRIM 
program.  You  will  receive  a notice  in  your 
microfiche  shipments  identifying  the  excep- 
tionally priced  reports  not  available  through 
SRIM. 

A deposit  account  with  NTIS  is  required 
before  this  service  can  be  Initiated.  If  you 
have  specific  questions  concerning  this  serv- 
ice, please  call  (703)  451-1558,  or  write  NTIS, 
attention  SRIM  Product  Manager. 


This  information  product  distributed  by 

U.S.  DEPARTMENT  OF  COMMERCE 

National  Technical  Information  Service 
t'285  Port  Royal  Road 
Springfield,  Virginia  22161 


...  _ 


DISCLAIMER  NOTICE 


THIS  DOCUMENT  IS  THE  BEST 
QUALITY  AVAILABLE. 

COPY  FURNISHED  CONTAINED 
A SIGNIFICANT  NUMBER  OF 
PAGES  WHICH  DO  NOT 
REPRODUCE  LEGIBLY. 


mm 

kl;.v  , 


184104 


UCLA-ENG-7067 
JANUARY  1076 


BSOCYBERNETIC  CONTROL  IN  MAN-MACHINE  INTERACTION: 
SEMI-ANNUAL  TECHNICAL  REPORT  1975-76 
(JULY  1,  1975  to  JANUARY  31,  1976) 


Sponsored  by 
Advanced  Research  Projects  Agency 
ARPA  Order  No.  3065 


i REPRODUCED  Br 

‘ NATIONAL  TECHNICAL 
1 INFORMATION  SERVICE 

U.  S DEPARTMENT  OF  COMMERCE 
SPRINGFIELD  VA.  22161 


D DC 


COMPUTER  SCIENCE  DEPARTMENT 

School  of  Engineering  end  Applied  Science 
University  of  California 
Los  Angeles 


DjimBUiivjM  statempnFX 
Approved  for  pubhc  release; 
£lftnbution  Unlimited 


SECURITY  CLASSIFICATION  OF  THIS  PACE  (Whnn  ».(«  F.ntand) 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 

1.  Rfc.RO  1 r NUMOfcN 

Z.  GOVT  ACCESSION  NO. 

3.  RECIPIENT'S  CATALOG  NUMBER 

» TITLE  (*nd  Subtitle) 

BIOC Y HERMETIC  CONTROL  IN  MAN-MACHINE 
INTERACTION:  SEMI-ANNUAL  TECHNICAL  REPORT 
(July  1,  1975  toJanua  ' 31,  1976) 

5.  TYPE  OF  REPORT  d PERIOD  COVERED 

Semi-Annual 

(July  1,  1975  to  J auu ary  31,  197 

6.  PERFORMING  ORG.  REPORT  NUMBER 

7.  AuTHOHfJj 

Jacques  J.  Vidal,  Principal  Investigator 
M.D.  Buck,  R.J,  Hickman,  R.H.  Olch 

B CONTRACT  OH  GRANT  NUMBERfV 

N00014-76-C-0185 

9.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

Department  of  Computer  Science 

School  of  Engineering  5 Applied  Science 

University  of  California.  Los  Anseles.  Calif.  90024 

10.  PROGRAM  ELEMENT.  PROJECT,  TASK 
AREA  * WORK  UNIT  NUMBERS 

ARPA  Order  Number:  3065 
S.  D.  State  Univ.  Foundation 

Sn nnnnfr'r>r*T  ■ O'PRnTG 

1 1.  CONTROLLING  OFFICE  NAME  ANO  ADDRESS 

San  Diego  State  University  Foundation 
San  Diego  State  University 
San  Diego  flaUfnrnis  9-VI.3  9 

12.  REPORT  DATE 

March  1976 

13.  NUMBER  OF  PAGES 

77 

u MONlToRiflu  AGENCY  NAME  » AODRESSfU  dllltrtnl  trim  Controlling  Olll ct) 

Office  of  Naval  Research 
1030  East  Green  Street 
Pasadena,  California  91105 

15.  SECURITY  CLASS,  (o  1 Ihlm  rtporl) 

Unclassified 

IS..  DECLASSI  FI  CATION/ DOWNGRADING 
SCHEDULE 

16.  DISTRIBUTION  STATEMENT  (o I Ihlt  Rtporl) 


DISTRIBUTION  STATEMENT  A 

Approved  for  public  release; 
Distribution  Unlimited 

17.  DISTRI  BUTIDN  STAT  EMEN  T (of  tba  abafract  antaead  In  Block  20,  If  dlftarant  from  Report) 


I «.  SUPPLEMENTARY  NOTES 

f v 

PRICES  SUBJECT  TO  CHANGE 

19.  KEY  WORDS  (Continue  on  ravaewa  aid*  II  nacaaaary  mnd  Identify  by  block  numbar)  — — 

BIOCYEERNETICS  VISUAL  PATTERN  STIMULATION 

MAN-MACHINE  COMMUNICATION 
EEC  CODES 

VISUAL  EVOKED  RESPONSES 

20.  ABSTRACT  (Contlnua  on  ravaraa  a Ida  If  nccaaaaty  arid  by  block  numbar) 

Semi-Annual  Report  is  a progress  report  of  the  UCLA  Biocybernetics  Control 
project  directed  toward  the  evaluation  and  implementation  of  man-machine 
command  and  control  procedures  that  incorporate  neuroelectric  signals  directly 
derived  from  the  brain. 

This  document  first  reports  on  the  current  state  of  the  project.  A first  milestone 
j has  been  reached:  one  such  man-machine  loop  has  been  operating  for  several 

1473  tomoN  or  I novas  soaiOLt'E  1 

s/n  01:1?  l f o i4  r.tioi  I 


DD  faDM 

I JAN  73 


(continued  on  back) 


SECURITY  CLASSIFICATION  OF  THIS  PAOEfHTun  D«l»  Enfrtd) 


20.  ABSTRACT  (Cont.) 

months  at  nearly  operational  performance  levels.  In  the  communication 
protocol  of  this  "MASTER-ROBOT”  team,  the  computer  robot  executes 
commands  encoded  in  the  master's  occipital  brain  waves  (as  SINGLE 
OPOCH  VISUAL  EVOKED  RESPONSES).  To  send  a command  the  master 
visually  selects  the  corresponding  command  symbol  from  a displayed 
set.  Symbol  pattern  aid  color  have  been  used  in  the  command  alphabet. 


SECURITY  CLASSIFICATION  OF  THIS  PAGE'WTi.n  Pal.  fnfr-d, 


BIOCYBERNETIC  CONTROL  IN  MAN-MACHINE  INTERACTION  * 
SEMI-ANNUAL  TECHNICAL  REPORT  19/5-1976 

JULY  1,  1975  to  JANUARY  31,  1976 


Jacques  J.  Vidal,  Principal  Investigator 


This  research  was  supported  by  the 
Advanced  Research  Projects  Agency 
of  the  Department  of  Defense 
Under  Contract  No.  NOOOl 4-/6-C-0 1 35 


School  of  Engineering  and  Applied  Science 
University  of  California 


Los  Angeles 


distribution  statement  a 

Approved  for  public  release; 
Distribution  Unlimited 


D D C 

BEEPS  nr? 

JUN  2?  1976 

EEEDTTE 

D 


I 


B I OC/BER  NET  I C CONTROL  IN  MAN -MACHINE  INTERACTION 

SEMI-ANNUAL  TECHNICAL  RERORT  19/5-19/6 
(July  I,  ly/a  to  January  31,  1976) 


Short  Title  of  Work 
AREA  Order  Number 
Name  of  Contractor 

Effective  Date  of  Contract 
Contract  Expiration  Date 
Amount  of  Contract 
Contract  Number 
Subcontract  Number 
Principal  Investigator 


Biocybernetic  Control 
306o 

The  Regents  of  the 
University  of  California 

July  I , | 975 

June  30,  19/6 

$110,000 

N000 I 4-76-C-0 I 85 
225076 

Jacques  J.  Vidal 
(213)  325-2358 


PERSONNEL 


Principal  Investigator 


Jacques  J.  Vidal 


Project  Manager 


Marshall  Buck 


Reasearch  Staff 


Randall  S.  Hawkins 
Robert  J.  Hickman 
Ronald  H.  Olch 


Students  and  Support  Staff  Dusan  Badal 

Thorn  Hertwig 
David  Holcomb 
Thomas  James 
Jeffrey  Johnson 
Larry  Sullivan 


V 


NEUrtOCYBEHNETIC  CONTROL  IN  MAN-MACHINE  INTERACTION 
SEMI-ANNUAL  TECHNICAL  REPORT  1975-1976 
(JULY  1,1975  to  JANUARY  31,1976) 
UNIVERSITY  OF  CALIFORNIA  AT  LOS  ANGELES 


CONTENTS 


Page 


ABSTRACT 


1. 1 ORIENTATION 
I . I . I His  tory 

1.1.2  Neurocybernetics  Paradigm 

1.1.3  Application  Range 

1.2  REVIEW  OF  RECENT  ACCOMPLISHMENTS 

1.2.1  Overview 

1.2.2  Data  Generation  and  Processing 
1.2.5  Bibliography 

2.  EXPERIMENTAL  PARADIGM 

3.  CLASSIFICATION  OF  BIOCYBERNETIC  SIGNALS 

EXPERIMENTAL  RESULTS 

3.  FACILITIES 

o.  BIBLIOGRAPHY 


Uj  U)  Uj 


ABSTRACT 


This  Semi  Annual  Report  is  a progress  report  of  the  UCLA  Biocy- 
ber-.etics  Control  project  directed  toward  the  evaluation  and 
implementation  of  man-machine  command  and  control  procedures 
that  incorporate  neuroelectric  signals  directly  derived  from  the 
brain. 

This  document  first  reports  on  the  current  state  of  the  pro- 
ject. A first  milestone  has  been  reached*  one  such  man-machine 
loop  has  been  operating  for  several  months  at  nearly  operation- 
al performance  levels;  In  the  communication  protocol  of  this 
41  MASTER-ROBOT"  team,  the  computer  robot  executes  commands  encod- 
ed In  the  master's  occipital  brain  waves  (as  SINGLE  EPOCH  VISUAL 
EVOKED  RESPONSES).  To  send  a command  the  master  visually  selects 
the  corresponding  command  symbol  from  a displayed  set.  Symbol 
pattern  and  color  have  been  used  in  the  command  alphabet. 
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l.l.  ORIENTATION 

I . I . I History 

The  present  document  is  the  Semiannual  Report,  covering  the 
period  June  I,  1975  to  January  31,1976  for  the  UCLA  Biocybernet- 
ics Control  in  Man  —Machine  Interaction  Project,  conducted  under 
ARPA  contract  to  the  San  Diego  State  Foundation  N000I4-76- 
C— 0 1 85  and  subcontract  # 225076.  The  long  term  goals  are  brief- 
ly reviewed  and  updated  below.  The  ultimate  goal  of  this  project 
is  the  evaluation  and  implementation  of  man— machine  command  and 
control  procedures  that  incorporate  neuroelectric  signals 
directly  derived  from  the  brain. 

1.1.2  Neurocybernetics  Command  and  Control  Paradigm 

The  use  of  bioelectric  signals  for  command  or  control  in 
the  man-machine  dialogue  can  best  be  discussed  under  a scenario- 
involving  man  in  the  role  of  operator  or  “MASTER"  (M)  communi- 
cating with  a ROBOT-SYSTEM  (RS)  The  RS  can  be  a sophisticated 
aircraft,  a computerized  command  system  or  a number  of  other 
man-machine  systems.  Quite  generally  it  can  be  said  that  com- 
munication between  M and  RS  takes  place  through  some  kind  of 
computer  terminal  albeit  probably  one  that  was  designed  espe- 
cially for  the  task.  An  aircraft  cockpit  or  the  control  console 
of  an  operation  room  are  essentially  computer  terminals  in  the 
present  context.  In  command  and  control  applications,  the 
master-robot  dialogue  relate  to  specific  events  occuring  in  the 
relevant  outside  WORLD  (W).  The  world  in  this  context  consist  of 
the  whoie  environment  that  affects  the  man-machine  mission.  It 
will  normally  include  the  physical  manoeuvering  space  needed  by 
the  man-robot  Learn  together  with  other  manned  or  automatic  sys- 
tems operating  within  the  same  space  in  various  relations  and 
capacities:  ( friend  or  foe,  active  or  passive  etc..)  Thus  W can 
be  a battlefield,  a theater  of  operations,  a trajectory  in  deep 
space  etc...  In  training  situations  iV  is  simulated  on  the  com- 
puter system,  together  with  the  RS  itself.  The  RS  assesses  '/I 
through  its  sensors  i.e,  as  a quantitative  set  of  numeric  or 
topical  variables.  The  master  observes  W using  visual .acoustic 
and  tactile  inputs  in  the  form  of  a set  of  displays  and  mes- 
sages combined  with  actual  frames  of  the  outside  world.  Thus 
his  input  will  include  some  of  the  sensor  data  as  shown  on  his 
instrument  panel  (while  other  sensor  data  will  be  kept  out  in 
automatic  control  loops.)  Both  partners,  the  man  and  the 
machine,  therefore  share  some  elements  in  their  input  sets  but 
operate  on  different  models  of  l».  These  models  are  central  to 
the  decision  strategies.  The  model  on  which  the  artificial 
intelligence  of  the  RS  relies  is  always  of  relatively  low  dimen- 
sion, limited  as  it  is  to  preassigned  classes  that  are  esta- 
blished in  terms  of  the  sensor  variables  /(J).  The  master  also 
relies  on  a model  of  the  world  but  in  the  form  of  a mental  per- 
ception of  toe  global  situation  that  is  or  immensely  larger 
dimensionality  and  capable  of  adaptation  to  unforeseen 
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^pnpr?nfr2  f‘fKZt  °n  senSory  capacities  and  on  a memory 

fhSfl1  ^ th®  same  time  less  precise  and  mere  adaptive  (than 
thS-  ?S  machiner/)*  also  benefits  from  the  unique 
capabilities  or  human  perception  and  intelligence  to  evaluate 

pattarns.  ,in  the  complete  context  of  the  known  situation 
and  to  reach  quick  decisions  between  perceived  alternatives. 

4-u  °f,a  mari-machina  team  is  tied  to  the  optimization 

iu,-^he  d V1!ion  of  lab?r’  limiting  the  human  burden  to  decisions 
lying  at  the  appropriate  level  of  abstraction  vfhile  placina  as 
many  ancillary  functions  as  possible  under  automatic  control. 

1.1.3  The  Application  Range  of  the  Neurocybernetics  Approach 
it  is  useful  to  recognize  the  instances  that  require  or 
the  search  for  neurocybernetic  channels  of  communication 
master  and  robot,  i.e.  the  supplemental,  remedial  or 
advantages  gained  by  adding  this  capability  to  the  robot 
Four  different  situations  appear  to  validate  the  neuro- 
approach,  because  of  conditions  that  prevent  or  limit 
normal  efferent  motor  channels  (such  as  voice,  key-" 


justify 
between 
unique 
system, 
cybernetic 
the  use  of 

board,  switch,  joystick,  light-pen  etc...)  These  situations  are 
re.pectiveiy  referred  to  as  Blocking,  Dysfunction,  Saturation 

^ the  availabiHty  of  the  information 

that  is  to  be  extracted  from  the  neuroelectric  signals* 

a)  Blocking:  Information  is  willfully  blocked  (stress  detector 

6i-!  information  is  sub-conscious  or  subliminal 
(psychotherapy,  eidetic  imagery, Tree  association,  memory  prob- 
ing, face  recognition  etc...).  H 

cr af n^rnmnTo ‘ A1i  ,mot°r  channels  are  busy  ( Hi-perf ormanc e 
craft  in  complex  combat  situation)  In  that  case  skeletal  output 

requirements  exceed  the  subject's  capacity  for  real-time  con- 

c)  Dysfunction:  Normal  motor  channels  are  disabled,  either  per- 
manently (prosthesis,  limb  control,  sensory  substitution  etc?..) 
or  temporarily  (hi -perf ormance  aircraft  in  hi-G  pull,  spacecraft 
occupants  in  low  metabolism  state,  zero-G  etc...) 

d)  Latency:  motor  channels  would  be  slower,  i.e.  have  longer 

reaction  times  than  that  provided  by  the  neurocybernetic  chan- 
nels emergency  control,  selection  among  set  of  countermeasures 
i.e.  alternative  subroutines  as  in  complex  command  and  control 
environment ) . 


Of  these  situations,  the  three  last 
master-robot  interaction. 


are  clearly  relevant  to 


UCLA-BC 


5 


3/26/76 


1 


1.2.  REVIEW  OF  RECENT  ACCOMPLISHMENTS 

1.2.1  Overview 

Definite  operational  success  has  been  reached  on  the  first 
phase  of  the  program  namely  the  correct  recognition  and  classif- 
lcation  of  stimulus  identity  in  a small  alphabet  of  possible 
stimuli,  in  single  epochs  of  EEG  visual  evoked  response.  Stimu- 
li used  are  flashes  either  patternless  in  a set  of  colors  oi* 
patterned  in  one  color.  Because  of  the  clear  differences  in 
codes,  patterns  and  colors  are  expected  to  be  identifiable 
separably  or  in  combination  although  no  experiments  with  combi- 
nations have  been  attempted  yet.  The  experimental  strategy  and 
the  real-time  data  processing  have  been  perfected  in  a succes- 
sion of  experiments  and  the  overall  approach  is  now  quite  suc- 
cessful with  almost  any  subject.  Expected  accuracy  with  random 
subjects  is  about  80%.  The  best  subjects  operate  consistently 


1.2.2  Data  Generation  and  Processing 

This  level  of  performance  has  been  obtained  by  submitting 
logi cal°order ) 9 Sequence  of  Processing  steps  namely*  (in  chrono- 

t Pr,i0/i  Artefact. Rejection*  This  step  that  takes  place 
before  actual  data  processing  of  the  epoch  is  most  important  in 
the  general  strategy.  The  frontal  pole  is  continuously  moni- 
tored for  excursions  beyond  normal  range.  The  real-time  experi- 
ment monitor  aborts  data  taking  when  such  activity  is  detected 
during  the  half-second  preceeding  the  actual  epoch.  If  it  ap- 
pears during  the  response  itself,  (after  stimulus)  acquisition 
is  completed  but  tne  epoch  is  rejected. 


1 .2.2.2  Wiener  Filtering*  This  is  an  optional  real- 
ing  (based  on  the  covariance  matrix  derived  from  the 
training)  .Its  function  is  to  optimize  the  signal— to 
rrom  the  standpoint  of  covariance  information.  The 
been  the  object  of  a communication  at  the  recent  IFI 
on  Optimization  in  Nice.  Wiener  filtering  is  soon 
bined  with  time-varying  bandpass  transformation  to 
components  at  the  beginning  of  the  epoch  (see  Fas 
ponents  -UCLA-Biocyberneti cs  - Final  Report  1975) 

1.2. 2. 3 Stepwise  Selection  of  Best  Samples*  The  ten  b 
pies,  from  tne  standpoint  of  discriminant  power,  are  se 
a stepwise  manner,  i.e.  by  order  of  decreasing  power, 
rive  ( electrode)channels  using  an  in-house  (BCD  versi 
now  classical  step-wise  discriminant  procedure  used  in 
cal  packages  such  as  the  new  Biomedical  P7M.  Th^  BCI  o 
interactive  and  designed  to  run  in  real-time  out~otherw 
give  identical  results.  Selection  is  performed  by  re 
calculating  r-ratios  , using  a training  set  containing 


time  filter- 
data  durinc 
-noise  ratic 
filter  ha£ 
P Conference 
to  be  com- 
enhance  fast 
t EEG  com- 


est  sam- 
lected  in 
from  the 
on  of  the 
statisti- 
rogram  is 
tse  would 
curs i vely 
ten  to 
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one  hundred  responses  for  each  stimulus  type. 

1.2. 2. 4 Recursive  outlier  rejection*  A linear  decision  rule, 
(Bayesian  model,  assuming  normalcy  of  distribution  and  identity 
of  variance  for  each  type)  is  obtained  on  the  basis  of  the 
selected  samples.  The  training  set  is  examined  for  outliers 
(i.e.  epochs  in  the  training  sets  that  the  decision  rule  mis- 
classifies  or  fails  to  classify  with  an  adequate  margin)  and 
such  epochs  can  be  removed.  The  program  then  returns  to  the 
stepwise  discriminant  selection  to  correct  the  decision  rule. 
The  recursive  outlier  rejection  scheme  has  not  yet  been  exten- 
sively tested  and  is  not  yet  implemented  on-line. 

The  decision  rule  is  calculated  using  all  non  rejected 
epochs,  and  applied  to  subsequent  epochs  as  they  are  collected; 
each  is  then  assigned  a probability  of  affiliation  to  each  of 
the  groups. 

1.2. 2. 5 Real-time  defaulting*  After  the  initial  processing  of 
the  training  set,  i.e.  in  subsequent  epoch-by-epoch  classifica- 
tion, the  former  outlier  rejection  is  replaced  by  the  confine- 
ment to  a "dorv't  know"  category  of  any  epoch  that  falls  beyond  a 
given  distance  margin  for  one  of  the  stimulus  types. 

1.2. 2. 6 Decision  Rule  Updating*  A recursive  “piggy-back"  pro- 
cedure is  available  as  an  option  for  on-line  experiments.  In 
that  case,  blocks  of  (usually  40)  epochs,  called  epoch  strings, 
are  sequentially  treated  as  training  sets  for  the  next  string. 
This  provides  a means  by  which  the  decision  rule  can  be  tracked 
as  it  undergoes  changes  due  to  task  learning,  operant  condition- 
ing or  any  other  cause. 

1.2. 2. 7 Neurocybernetic  loop*  The  real-time  classification  of 
evoked  responses  has  finally  been  incorporated  in  an  actual 
man-machine  communication  scenario.  In  this  scenario  the  master 
is  required  to  run  a maze  displayed  on  a graphic  terminal.  The 
moving  target  in  the  maze  (the  "mouse-"  or  "mobile")  is  directed 
by  visually  acquiring  (i.e.  directing  the  gaze  to)  one  of  four 
visual  "keys"  that  frame  the  field  of  operation  and  thus  signal 
"up",  "down",  "left"  or  "right".  A diamond  shaped,  checkerboard 
pattern,  that  appears  briefly  between  the  keys,  produces  the 
evoked  response,  with  the  encoded  information. 

Each  stimulus  type,  as  soon  as  it  has  been  identified  in  the 
(occipital)  EEG  response  causes  the  system  to  implement  the 
move?  thus  each  successful  move  constitutes  reward.  The 
resulting  operant  conditioning  scheme  is  therefore  directed  to 
the  quality  or  accuracy  of  the  classification  in  a non-specific 
way.  That  learning  takes  place  can  be  seen  in  the  results  of 
successive  runs  in  the  piggyback  mode.  Typically  an  increase  in 
performance  is  recorded  over  the  two  or  three  first  runs  and 
will  stabilize  afterward.  The  nature  of  that  learning  is  not 
elucidated  yet,  but  since  these  experiments  clearly  deal  with 
"exogenous"  components  of  the  evoked  response  which  are  presumed 
to  be  relatively  resistant  to  operant  conditioning  procedures, 
it  is  reasonable  to  suggest  that  the  increase  in  performance  is 
probably  due  to  the  avoidance  of  interfering  processes  such  a 
muscle  artefacts  or  even  the  blocking  out  of  interferring  mental 
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activity  (casual  evidence  for  the  latter  is  found  in  the  verbal 
reports  obtained  from  the  subjects  in  recounting  their  percep- 
tions during  experiments) 

The  maze  experiments  have  been  the  object  of  a presentation  at 
the  recent  IEEE  Symposium  on  Man  Machine  and  Cybernetics  in  San 
Francisco. 

With  regard  to  these  experiments  and  the  entire  first-phase 
of  the  project  it  is  felt  that  a limit  has  been  reached  in  the 
procedure  . No  radical  change  in  the  data  processing  approach 
will  be  necessary  to  maintain  high  levels  of  performance  over  a 
large  range  of  similar  experimental  conditions,  and  we  would 
venture  to  say,  with  acoustic  or  tactile  stimuli  as  well.  The 
relatively  minor  changes  that  are  now  in  the  works  and  in  par- 
ticular the  time-varying  Wiener  filter  are  expected  to  add  a few 
percents  to  the  current  levels.  In  addition,  new  low  noise  EEG 
preamps  are  expected  to  increase  the  performance  of  the  subjects 
whose  brain  signals  were  smaller  than  average,  therefore  some- 
what buried  in  the  instrumentation  noise.  Present  amplifier 
noise  is  I to  I .5  microvolts  peak  to  peak?  the  new  amplifiers 
will  be  better  by  a factor  of  five  in  noise,  and  60  db  better  in 
common  mode  noise  rejection.  Thus  the  first  phase  of  the  pro- 
ject (time-locked  visual  evoked  responses  with  stimuli  that  are 
(easily)  subject  discriminable  and  exogenous  signatures  in  the 
EEG)  is  to  be  considered  terminated. 
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CHAPTER  2 

EXPERIMENTAL  PARADIGM 


2.J  -Mac  h me  £nvir cnmer  t 

Man  communicates  ty  perceiving  his  environment  and 
effecting  a structured  response  tc  it.  Tne  structure  is 
reflecteu  in  an  ordered  selective  attention  toward  each 
particular  component  of  tne  perception,  along  with  an 
interactive  protocol  established  for  information  transfer 
with  his  environment.  This  research  examines  responses  to 
inputs  on  the  visual  and  auditory  mouaiities.  The  responses 
observed  are  electrical  fluctuations  sensed  on  the  surface 
of  the  seal*..  These  signals  represent  a simple 
non-interfering  scheme  to  observe  the  early  components  cf 
the  biocyoer retie  processes  of  the  human  nervous  system. 

The  paradigm  to  examine  and  demonstrate  man-machine 
ccmmunicaticr  has  been  implemented  in  tne  Drain  Computer 
Interface  laboratory.  (A  current  description  of  this 
facility  as  it  applies  tc  this  research  is  provided  in 
Appendix  I.)  The  paradigm  can  be  visualized  through  a 
scenario  in  which  the  principal  functions  arc  performed  by 
three  elements,  the  man,  the  machine,  ana  the  i nvironment. 
Tne  rollo-rfirg  notation  and  functional  interactions  provide  \ 
description  ct  t.ie  role  j 1 these  elements  in  the*  paradigm. 
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1) 


2) 


3) 


Man  as  master  { M ) the  master  observes  a robot 
immersed  in  a world  and  learns  to  generate  messages, 
according  to  constraints  imposed  by  the  world,  and  the 
robots  capacity  for  understanding,  so  that  the  robot 
responds  to  the  master's  goal. 


The  machine  as  a 

robot 

(H)  --  the 

robot 

learns  to 

interpre  t 

messages 

from 

the  master 

and 

behaves 

according 

to  its 

awareness  of  the 

world 

and  the 

constraints 

imposed 

by  the 

world. 

The  environment  as  the  world  (m)  — the  world  defines 
the  universe  of  discourse  in  the  communication  between 
master  and  the  robot  and  constrains  their  behavior. 


COMMUNICATION  AND  fEEDBACK  CONTROL 


Figure  2-1  presents  a simplified  block  diagram  of  a 
fciof eedtack  control  loop  employed  in  the  experiment.  The 
robot  and  the  world  are  simulated  by  the  computer  system  and 
their  irteraction  is  displayed  to  the  master,  a human 
subject,  by  a cathode  ray  tube  (CRT)  display.  The  subject 
visually  observes  the  rocct's  behavior  as  displayed  on  the 
CRT,  and  issues  messages  to  control  the  rcoot  according  to  a 
protocol  with  the  computer  system.  This  computer  system 
simulates  the  robot  and  world  with  a data  collector,  message 
classifier,  and  robot  simulator.  The  data  collector  and 
message  classitier  reflect  the  world's  constraints  on  the 
robot's  perception  and  interpretation  or  the  master’s 
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messages,  w h 1 x e t he  n o b o t simulator  implements  the  dccisiors 
mad*  according  to  the  world's  constraints  on  its  behavior. 

The  EEG  biofeedback  exhibited  in  this  research  has 
different  objectives  anu  implementation  than  previous 
tiofeedcack  experiments  of  this  type.  in  this  work, 
biofeedback  is  an  element  in  a man-machine  dialog,  rather 
than  simply  an  indicator  of  behavior  modification.  The  role 
of  the  robot  is  central  to  the  understanding  of  this  concept 
of  feedback  controlled  communication.  It  is  through  the 
behavior  of  the  robot  that  the  master  is  able  to  determine 
that  his  implied  commands  are  getting  through,  as  the 
behavior  of  the  robot  constitutes  the  feedback  signal. 

The  data  processing  capability  cf  the  brain  Computer 
Interface  laboratory  permits  analysis  of  single  epochs  of 
multi-channel  2EG  data,  and  the  techniques  developed  can 
identify  (cn  a relatively  medal-free  basis)  amplitude, 
frequency,  and  phase  attributes  of  the  neuroelectric  signals 
involved  in  the  messages  transmitted  tc  the  robot.  This 
provides  a substantial  variation  and  expansion  from  earlier 
techniques  wnere  simple  filters  were  used  to  detect  cyclic 
amplitude  uetavior  xn  E EG  signals. 


iai  Current  Experimental  Implementation 

.he  initial  scenario  reaturas  a (rooot)  mouse 
attempting  tc  free  itselr  from  confinement  in  a maze.  The 
robot  mouse  must  move  up,  down,  left,  or  rignt  along  the 
channels  of  a maze  wnile  attempting  to  avoid  contact  with 
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I. 


the  walls  or  unnecessary  travel  down  blind  alleys.  Figure 

2-2  presents  a diagram  of  a raze  in  which  the  mouse  is 

initially  confined  at  the  center.  The  shortest  path  to  the 
exit,  in  tne  lower  left  corner,  contains  an  equal  number  cf 
moves  in  each  direction.  The  experimenter  can  vary  the 

mouse's  perception  of  its  wcrla  with  several  options  that 
reflect  different  levels  of  intellectual  capability  in  the 
robot.  For  instance,  the  simplest  option  depicts  a dumb 
mouse  that  pays  no  attention  to  the  proximity  of  adjacant 
walls,  and  has  no  preference  to  any  particular  move  prior  to 
receiving  a message.  Other  options  allow  ror  smarter  robots 
that  ace  aware  of  adjacant  walls  or  previous  moves.  In 

addition  to  illustrating  the  value  of  the  robot's 
intelligence  in  this  approach,  these  options  provide  a 
mechanism  for  the  experimenter  to  adjust  the  requirements  on 
the  master's  message  generating  capacity.  This  provision  is 
invaluable  in  letting  the  human  subject  gradually  develop 
his  neurocy bernetic  control  capacity. 

lhe  major  constraints  on  the  communication  between 
master  and  robot  can  be  illustrated  by  describing  the  time 

sequence  of  the  uata  collection  and  command  classification 

v 

processess.  Figure  2-3  presents  a simplified  diagram  of  the 
activities  and  events  involved  in  a single  cycle  of  the 
communication  protocol.  This  dialog  protocol  is  the  basis 
for  the  analysis  of  the  data  epochs  as  a time  synchronous 
exchange  between  the  master  and  tue  data  collector.  The 
initial  event  of  the  cycle  is  a ready  cue  follow  id  by  a wait 
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interval  (0.8  sec)  to  allow  the  master  to  select  a command. 
The  ready  cues  include  a time  coincident  beep  and  LED 
illumination  at  one  or  all  vertices  of  what  at  stimulus  time 
will  appear  as  a diamond  shaped  checkerboard  pattern 
superimposed  on  the  CRT  display.  A strobe  light  provides 
the  interrupt  stimulus,  impressing  the  checkerboard  pattern 
on  a portion  of  the  retina,  depending  on  which  LED  the 
master  directed  his  gaze  during  the  wait  period.  Figure  2-4 
presents  a diagram  of  the  superimposed  checkerboard  and  LED 
display,  and  also  the  form  of  this  pattern  as  it  would 
appear,  given  fixation  at  each  vertex.  The  visual  evoked 
response  to  the  stimulus  is  transmitted  from  the  master  to 
the  robot  where  it  is  classified  to  provide  the  command. 
The  robot  simulator  carries  out  the  command  and  moves  the 
mobile  accordingly. 

In  the  subseguent  "verify'*  phase  the  changing  display 
constitutes  a stimulus  tc  the  master  that  confirms  cr 
intirms  the  successful  conclusion  of  the  command.  The 
resulting  eveked  response  can  then  be  transmitted  to  provide 
a "verify"  message  that  can  either  cancel  the  last  command 
or  modify  the  robot's  tias  in  the  next  cycle. 

Ihe  nature  of  scalp  electrical  activity  necessitates  a 
somewhat  involved  logic  to  test  the  properties  of  the 
signals  during  each  activity  of  the  cycle.  Artifactuai 
signals  of  apparent  random  cccurance  can  mask  desired 
signals  and  must  be  identified  and  separated.  The  artiracts 
can  result  from  such  influences  as  muscle  activity,  ^ye 
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movements  or  blinks,  alpha  waves,  anu  sleep  spindles.  The 
amplitude  of  these  signals  can  te  several  times  those  cf 
normal  responses  while  the  frequencies  can  vary  from  1-2 
hertz  for  eye  artifacts,  8-13  hertz  for  alpha  and 
spindles,  and  50  - 75  fcr  muscle  activity. 

2.4  Initial  Training 

Before  the  master  can  exercise  control  of  the  robot,  an 
initial  set  ct  the  signals  must  be  analysed.  The  process  of 
developing  a reliable  channel  of  communication  involves 
adaptation  or  the  decoding  aigcritnms  implemented  by  the 
computer.  This  initial  phase  is  then  complemented  by 
bioteedback  as  master  experiences  the  etfects  of  changes  in 
his  signal  generation.  Cn-liue  real-time  data  processing 
techniques  are  employed  tc  achieve  this  biofieedtack 
learning. 

Thus  the  sequential  procedure  begins  with  an  epoch 
string  that  involves  collection  of  an  initial  set  of  command 
data  massages.  This  set.  is  analysed  to  produce  a 
classi f icat ion  function  (CF)  which  is  then  employed  to 
sequentially  classify  each  command  message  generated  in  the 
next  epoch  string.  This  subsequent  phase  involves  data 
collection,  c cm maud  classification,  and  activation  of  the 
robot  b a s <cd  cn  the  results  ct  the  classification.  The  robot 
action  , CDSttved  by  the  master,  constitutes  the  feedback 
signal  and  subsequently  becomes  the  stimulus  cor  the  verify 
message.  T ne  command  messages  ore  then  combined  with  those 
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collected  in  tne  tirst  epoch  string  to  produce  an  updated 
Cl,  that  incorporates  the  effects  of  training  and 
adaptation.  Similarly  the  verify  message  sat  from  the 
S-Ccnu  epoch  string  can  re  analysed  to  produce  a 
classif icat ion  function  (VF)  which  will  then  be  employed  to 
subsequently  verity  command  interpretation  in  the  next 
phase.  This  process  is  repeated  through  several  epoch 
strings  with  the  data  collected  in  each  phase  refining  the 
CF. 

, * k 

2^5  Eat  a Collection  Protocol 

The  detailed  protocol  between  the  master  and  the 
on-line  data  col] actor  is  illustrated  in  Figure  2-5.  This 
diagram  presents  sequentially  the  segments  of  the  data 
collection  cycle  and  includes  elements  of  the  controlling 
logic  lor  data  aeguisitier  and  artifact  handling.  This 
enables  the  master  to  sychronize  his  function  with  the  data 
collection,  classification,  display,  and  storage  functions 
that  the  data  collector  performs.  The  presentation  of 
Liofeedtack  signals  to  the  caster  provides  two  tyoes  of 
information.  Ihe  tirst  is  characterized  by  the  cathode  ray 
display  provided  by  the  computer  and  presents  the  results  of 
the  classification  of  tne  command  signal,  i.e.,  the  action 
of  the  robot.  Ihe  second  signal  is  provided  by  a status 
panel  consisting  of  a series  ci  colored  lights.  These 
lignts  display  the  present  status  o l.  the  data  collector.  A 
red  light  indicates  the  defection  of  an  artifact  in  the 
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datr~ . This  signal  will  remain  for  a short  interval 
delaying  the  restart  of  the  collection  cycle  and  eventually 
allowing  the  master  to  alter  his  behavior  in  the  direction 
ot  artifact  reduction,  while  the  data  ccllectcr  restarts  the 
real-time  data  collection  segment.  Thus  the  master  is 
trained  to  produce  si,jnals  that  pass  the  artifact  filter. 
The  design  cf  this  artifact  detection  filter  is  an  important 
aspect  or  the  experimental  paradigm  and  will  be  discussed 
later.  The  Figure  also  illustrates  the  sequence  of  tests 
perfcraeu  tc  identify  artifacts.  These  tests  are  performed 
at  the  beginning  of  an  epoch,  after  completion  of  the 
command  message,  and  after  the  verification  message.  If 
artifacts  are  detected  at  the  first  two  tests,  the  epoch  is 
afcandoneu  ana  rescheduled  after  a short  delay  with  the 
response  of  the  rotct  being  delayed  until  the  next  epoch. 
An  artifact  during  the  verification  results  in  "ao  decision” 
regarding  the  verification.  The  epoch  is  not  included  in 
future  CF  generation  but  the  response  of  the  robot  is 
handled  according  tc  the  cpticns  available  in  the  experiment 
ccntrcl  parameters. 

2^6  B~c teed back  displays 

The  main  bicf feedback  signal  to  the  master  is  provided  by 
the  catnoue  ray  tube  which  uisplays  the  world  model  and  the 
robot's  actions  in  this  man-machine  paradigm.  Figure  2-2 
presents  an  example  of  such  a display . The  robot  in  this 
example  acts  on  a ucnile  , "the  mouse”,  displacing  its 
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position  in  a two-dimensional  maze.  The  commands  invoke 
changes  in  position  cf  the  motile.  Namely,  move  up,  move 
down,  move  left,  or  move  right.  The  numbers  appearing  on 
tne  diagram  represent  decision  points:  the  mobile  waits  at 
a decision  point  for  receipt  of  a command  which  defines  the 
direction  in  which  it  is  to  proceed.  The  mobile  then  moves 
in  this  direction  at  a predetermined  rate  and  halts  at  the 
next  decision  point  and  waits  for  the  next  command  from  the 
master. 

The  alphanumeric  characters  appearing  at  the  top  of  the 
display  denote  cpticr.al  mcdes  of  behavior.  The  decision 
node  restricts  the  mobile  to  move  only  to  an  adjacant 
decision  pcir.t,  unless  it  hits  an  adjacant  wall  in  which 
case  it  returns  to  the  present  decision  point.  The  wall 
mode  lets  the  mobile  move  until  it  strikes  a wall  and  then 
stops  at  the  decision  feint  near  the  point  or  impact.  This 
mode  involves  fewer  decision  feints.  Upon  reaching  the  end 

point  (x)  the  mobile  is  automatically  returned  to  the 
starting  point. 
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3.  CLASSIFICATION  OF  3IOCY  BERN  STIC  SIGNALS 

♦ 

Ccnven  tional  statistical  analysis  techniques  have  been 
employed  to  classify  single  epochs  of  biocybernetic  data  to 
one  of  a set  of  groups,  where  each  group  is  associated  with 
a specific  command  to  the  machine  under  control.  The 

principle  questions  invclved  in  the  mathematical  treatment 
of  the  data  epochs  includes  the  choice  of  a decision  logic, 
a mathematical  model  of  the  statistics  or  the  data,  and  a 
procedure  to  efficiently  identify  the  components  of  the  data 
epoch  that  provide  tne  most  information.  Bayes  theorem  cn 
•inverse*  cr  posterior  probability  in  combination  with 
stepwise  discriminant  analysis  techniques  as  described  by 
Dixon  form  the  basis  of  the  approach.  The  decision  logic  is 
based  on  the  Bayesian  technique  which  combines  prior 
information  with  transmitted  information  to  produce 
posterior  probabilities  for  group  classification  of  the 
e^och.  Ihe  term  posterior  denotes  the  .act  that  the 
prctabiii ties  arc  determined  after  incorporating  data 
extracted  from  the  present  epoch.  The  posterior 
probabilities  are  unique  tc  the  present  epoch  and  represent 
maximum  likelihood  estimators  of  the  percentage  of  these 
types  of  epochs  that  can  bf  found  in  each  group.  The 
statistics  cr  the  data  epochs  are  characterized  by 
multivariate  normal  functions  witn  different  meins  for  <jich 
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group  and  a covariance  matrix  that  is  constant  over  the 
groups.  The  Fisher  F-statistic  for  group  separation  is  the 
measure  employed  to  evaluate  the  individual  components  cf 
the  data  epochs  and  a stepwise  procedure  selects  a subset  of 
eroch  variables  that  prcviue  a good,  but  rot  necessarily 
optimal,  scheme  for  extracting  the  signal  components  from 
the  data  epoch. 

Tne  classification  of  the  epoch  is  accomplished  by 
assigning  the  epoch  tc  the  group  with  the  maximum  posterior 
probability,  given  that  this  posterior  probability  exceeds  a 
specified  threshold.  An  assignment  tc  a group  is  not  made 
when  the  maximum  posterior  probability  is  less  than  the 
threshold.  Bather,  the  epoch  is  placed  in  a ’default' 
category  that  results  in  no  decision  as  far  as  the  system  is 
concerned,  tpcchs  placed  in  the  default  category  may  then  be 
excluded  trom  use  in  determining  subsequent  discriminant 
functions.  This  approach  attempts  to  isolate  those  epochs 
that  are  apparently  not  representative  of  the  cet  upon  which 
the  present  discriminant  function  was  determined.  While 
this  approach  may  appear  to  favor  the  'status  guo'  it  will 
accept  change,  as  lcng  as  the  threshold  is  not  too  high  and 
the  changes  are  gradual  and  do  nor.  produce  a set  of 
posterior  pccfcaDilities  that  are  all  less  than  the 
tar isholu. 

The  prior  information  represents  what  was  known  before 
tne  present  epoch  was  received  and  incorporation  of  this 
prior  information  into  the  classi  ric  ltior.  function  modifies 
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the  inttr pretation  or  the  ccnmands  generated  by  the  subject, 
and  thereby  shapes  the  behavior  of  the  man-machine  system. 
Knowledge  or  the  state  of  the  machine  just  prior  to  the  time 
ot  the  epoch  is  used  to  weight  the  chance  that  each  of  the 
possible  commands  are  encoded  in  the  epoch.  Thrs  effect  is 
characterized  as  a measure  of  local  context  and  varies  as  a 
function  of  the  state  of  the  machine.  Thus  the  command 
generating  capacity  of  the  subject  achieves  "apparent" 
additional  dimensions  cf  control  in  the  form  of  special 
behavior  at  particular  conditions.  The  following  sections 
provide  a development  of  the  functions  used  to  implement 
these  techniques. 

PCSTEJ1C5  PBCEAEILITILS 

The  posterior  probabilities  constitute  a set  cf 
mutually  exclusive  estimates  or  the  chance  that  a given 
epoch  should  be  associated  with  each  of  the  commands.  These 
protaoilities  are  determined  from  the  3ayes  equation  where; 


r ( w ) ( X | w ) 

i i 

F(w  |X)  - 


P ( w ) p ( X | w ) 
x k 


(3.  1) 


where: 
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x 


a 'd*  dimensioned  data  vector 
extracted  from  an  epoch. 


P (w  ) 
i 


F (X|«  ) 
i 


pricr  probability  of  group  w . 

i 

conditional  probability  data  vector  X 


is  from  group  w . 

i 


PRIOJj  PRCiLfcEILIXIES  „ 

The  uesired  command  for  the  machine  under  control  can 
be  highly  correlated  with  the  state  of  the  machine  at  any 
particular  time,  and  knowledge  of  this  correlation  provides 
the  tasis  fcr  determining  the  prior  probabilities  for  the 
Bayes  eguaticn. 

The  pricr  probabilities  are  determined  for  the  maze 
experiment  according  tc  several  schemes.  The  first  method 
involves  examining  the  space  auout  the  mouse  as  it  reaches 
different  positions  in  the  maze,  ana  assigns  lew 
probabilities  to  commands  that  would  move  the  mouse  directly 
into  an  augacant  wall.  A second  method  is  designed  for  the 
'wail'  moue,  where  the  mouse  proceeds  down  a channel  until 
it  strikes  a wail.  This  method  determines  prior 
probabilities  at  each  position  in  tue  maze  suen  that  one  cr 
three  commands  is  possible,  back- tracking  tne  previous 
command,  or  either  direction  orthogonal  to  the  previous 
command.  Continuing  in  the  same  direction  as  tna  previous 
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command  is  net  desirable  as  the  previous  command  will 
position  the  mouse  at  the  end  of  the  channel.  Back- tracking 
is  weighted  according  to  the  probable  error  in  the  previous 
command,  while  the  couplement  of  this  error  is  divined 
evenly  to  weight  the  two  crtnogonal  directions.  A final 
method  represents  a control  comparison  and  assumes  that  no 
information  is  available,  thus  the  prior  probao ilities  for 
each  command  are  equal.  Tne  following  equations  define  the 
prior  probabilities  for  the  different  schemes  in  the  maze 

paradigm. 

1)  Smart  mouse: 

/0,  ir  path  i is  obstructed. 

( 

p(»  i •<3-21 

ij 

V 1/n  , if  path  i is  cle^r. 

j 

w here: 

j = a position  inaex  in  the  mace, 
n = the  number  ct  clear  paths  at  j. 

j 


2)  Blind  mouse: 


P(w  ) 

i-  j 


f 0. 


continuing  the  command  from  j-1. 


L 1 - P ( I X) 

i j-1 


Piw  IX)  / 2, 
i j-1 


]#  back- tracking 
command  at  j- 
either  command 


the 

• ( 3.  3) 


orthogonal  to  j-1. 


J)  Dumb  mouse: 

P («  ) = 1/4,  for  all  patns  at  j.  (3.4) 

i 1 

It  is  possible  tc  define  many  such  schemes  to  temper 
the  results  of  pattern  classifications  with  prior 
information.  Those  just  described  produce  behavior  where  the 
mouse  exhibits  some  partial  knowledge  of  the  proper  command, 
based  on  the  context  of  the  position  in  which  it  finds 
itself,  and  the  command  generating  capacity  of  subject  can 
be  selectively  enhanced  without  requiring  additional 
dimensions  ct  control. 

££N£IiIQNAL  EPCBASILITIJIS 

The  conditional  probabilities  are  determined  directly 
from  each  epoch  ct  b iocy be r na tic  data  generated  by  the 
subject.  These  probabilities  represent  measures  of  the 


percentage  ci  epochs  in  each  ot  the  groups  that  are 


-ncodod 
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like  tlic  data  received  Lrem  the  subject  in  this  epoch. 

Ihe  biccy oerne tic  data  extracted  from  an  epoch 
constitutes  a vector  which  is  a suD-set  of  the  samples  cf 
ELG  voltage  iluctuaticr.s  that  comprise  a data  epoch.  The 
step-wise  procedure  employed  to  select  the  vector  components 
from  the  data  epoch  will  be  described  later.  The 
multivariate  normal  distribution  is  a convenient  candidate 
for  a medal  ct  the  statistics  cf  the  data  vector.  This 
distribution  runcticn  is  defined  by  the  followinj  equation. 


t -i 

exp  [ -0.5  (X-U  ) (S  ) (X-U  ) J 
i i i 


f ( X | w ) 


i 


d V2 

[ (2  ) IS  1 j 

i 


(3.5) 


w h e r e : 


U = mean  vector  from  9 roup  w . 
i i 


S = covariance  matrix  or  vectors  in  g roup  w . 

i i 

A significant  reduction  in  computational  complexity  and 
data  processing  recui cements  can  he  achieved  if  the 

ccvuLianeos  of  the  multivariate  distribution  functions  for 
each  classification  group  are  assumed  to  ce  e q ml.  This 
mcdei  with  the  assumption  ct  c-.ju  >1  covariances  among  ;roups 
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is  not  inccnsistant  with  studies  of  the  nature  ot  EEG 
amplitude  statistics.  The  adequacy  of  this  simplified  model 
in  characterizing  the  nature  of  the  biocybernetic  data 
vector  is  presented  in  the  discussion.  The  equations  fee 
the  ccnditicnal  prooabili ties  with  this  simplified  model  are 
thus: 


t -* 

exp  [-0.5  ( X- U ) S (X-U  ) ] 
i i 


P ( X 1 w ) 


l 


d V2 

C (2  ) I S | ] 


d I X | (3.6) 


Incorporating  this  model  into  the  posterior  probability 
equations  and  eliminating  terms  which  are  independent  ot  the 
classification  groups,  produces  the  following  classification 
functions. 


t -» 

P(w  ) exp[  -0 . 3 (X-U  ) 3 (X-U  ) ] 


i i i 

p(w  |X)  = - (3.7) 

i 

t - 1 

p(w  ) e xp[  -0 . 5 (X-U  ) 3 (X-U  ) ] 

X k k 


Expanding  tht  exponentiated  term,  jives: 
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t - 1 t - 1 t-»  t - 1 

L • J = -0.5  (X  S X-US  X - X S (J  + U S U)  (3.8) 

i i i i 

Tiie  first  tern  is  independent  of  the  classification  yrcups 
auu  can  Le  eliminated  frcrn  the  numerator  and  denominator , 
and  the  second  and  third  terms  produce  identical  terms  upon 
expansion , sc  that  the  exponentiated  term  becomes: 


[.] 


t - 1 

-0.5  (US  U ) 
i i 


t - 1 

+ U S X 
i 


which  can  be  rewritten  as: 


(3.  y) 


l • ] - a + BX 

i i 


(3.  10) 


where: 


t -» 

B = U S 
i i 


a - - 0 . 5 ( B l j ) 
i i i 


and  the  posterior  prcuauilities  reduce  to: 
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p ( w ) e X p ( d +IJX) 
i i i 

p (w  IX)  = . (J.  1 1) 

i > 

p ( ) e x p ( a ♦ 3 X ) ’*  < 

x k k 


Or  VjiRI A B L t S 

The  first  step  toward  selecting  a set  or  variables  from 
d LiocyLernetic  data  epoch  is  the  development  ot  d criteria 
tor  measuring  the  relative  pericnaance  of  the  variables  in 
separating  the  classi tica tic n groups.  The  criteria  employed 
in  this  research  is  based  cn  the  Lamiliar  technique  cf 
Fisher  which  examines  the  ratio  of  sums  or  square  deviations 
ot  the  within  group  means  about  the  grand  mean  to  the  sums 
of  square  deviations  about  the  within  group  means.  The 
assumption  cf  e^ual  covariances  among  the  groups  provides 
for  the  ueteroiinaticn  ot  an  F-statistic  for  sets  cf 
variaoles  from  a population  ct  epochs.  The  significance  cf 
tne  gtcu£  reparauility  of  tun  wit  air:  group  means  or  these 
variaties  can  then  ts  tested  against  the  chance  aspects  cf 
the  variation  c:  these-  group  means. 

1 • * 

Tne  gtr.cLdl  solution  rcc  t;u:  set  or  variaoLos  that 

maximize  the  separation  or  the  groups  involves  the  solution 
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Tae  variances  tor  the  sub-set  ot  variables  are  determined 
from  a selection  vector  and  the  sums  cf  squares  and 
cross-proaucts  matricies  of  all  the  variacles  in  the  epoch. 
Two  aiatricies  are  denr.ea,  the  first  is  for  the  within  group 
means  about  the  population  grand  mean  and  the  second  is  fcr 
the  variaules  about  the  appropriate  witnir.  group  means. 
These  iratricies  are  defined  as  follows: 

t 

as  = (U  - £J  ) (U  - U ) (3.  12) 

p k p x p 

where: 


Sb  = the  sums. ot  squares  ana  cross-products 

y 

matrix  for  the  within  group  means 
acout  the  population  grand  mean. 

E = a vector  or  ali  variables  in  an  epoch. 

U = the  within  group  mean  of  E. 

y 

U = the  population  jrai.i  moan  or  1. 

V 

Similarly,  the  within  groups  sins  or  snares  and 
cress- products  matrix  cr  the  uata  vector  is  a Lined  i y: 
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t 

SS  = (£  - U ) (1  - 'J  ) (3.13) 

k k k 


The  F-statistic  for  a particular  sub-set  ol  variables 
can  be  defined  as: 


F 


t 

(n-g)  V SS  V 
f 


t 

(g-1)  V SS  V 

K 


(3. 14) 


where: 


n - nusrfcr  of  epochs. 


g = number  of  ciassir ication  groups. 


V = a vector  with  arbitrary  binary 

elensents  rcr  selecting  variables, 
1 tc  select,  0 to  reject. 


Ihe  selection  ci  the  set 
maximizes  the  above  F-ratic 
differentiation  of  eg.  3.1b  with 
vector  V ina  equating  the  result 


of  epoch  variables  *-hat 
can  be  achieved  through 
respect  to  the  selection 
tc  zero,  giving: 
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t t 

F 2 (n-q)  (V  S3  V S3  V - V S3  V SS  V) 

K F P k 


= U (3.1b) 


t 2 

(g- 1)  (v  ss  v) 

k 


2(n-g)  (SS  V - Z S3  V) 
P k 


(g-1)  V 33  V 

k 


where: 


(3.  1c.) 


t 

V S3  V 
P 


t 

VS 3 V 
k 


(3.  17) 


The  solution  oi  eq.  3.  It  is  equivalent  to  the  solution 
or  tne  rcllcwinq  system  oi  linear  equations: 


- x 

(Si  33  - z I)  V = 0 (3.  H) 

k i 

The  stef. wise  procedure  ci  Erroynsen  is  employee  to 
systematically  solve  these  equations  and  thereby  select  the 
vutiaulcs  1LC.I  the  epoch,  .nis  t- chm  iu*.  also  te  ruinates  the 
process  c.ei;  the  sigm  r ica  net  of  the  contribution  ct  all 
v a tier  1*  a ret  > n tolecteu  fails  below  a ctf  s »t  level.  is 
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procedure  tests  variables  in  the  selected  set  (it  any)  ter 
removal  iron:  the  set,  ana  then  tasts  variables  that  have*  net 
yet  been  selected  for  inclusion  in  the  set.  The  ^’-statistics 
or  the  indi v iaual  varieties  are  compared  to  F- to- remove  or 
F-tc-enter  levels  tc  determine  the  course  of  the  stepwise 
selection.  The  process  is  terminated  when  all  variables  not 
yet  included  in  the  set  possess  F-statistics  less  than  the 
F- to-en tt r level. 

ihe  stepwise  technique  employs  the  Gauss  elimination 
method,  as  described  ny  Crden  , for  the  solution  or  the 
simultaneous  equations.  Initially  the  variable  associated 
with  the  laryest  F-statistic  is  selected,  given  that  it 
exceeds  the  F-to-eriter  criteria.  This  variable  is 
eliminated  iron  the  matrix  by  the  ramiliar  pivotal 

transform.it  ns  where  the  elements  a of  the  succeeding 

ij 

matrix  are  generated  by  riveting  the  present  matrix  around 
the  selected  variable,  unnoted  here  by  the  subscript  kk. 
Tne  rules  rot  updating  the-  matrix  are  summarized  as  follows: 
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d 

ij 


a -aa  /a 
ij  i k k 3 kk 

a /a 
kj  kk 


-a  /a  ; 
ik  kk 

1 / a ; 
kk 


if  i f k,  j * k 

if  i = j,  j * k 

(3. 19) 

if  i * k,  j = k 

if  i = j,  j = k 


lyrically,  the  variables  in  the  Liocybernetic  data 
epochs  are  correlated  over  intervals  of  several  adjacant 
variables,  that  is  neighboring  variables  tend  to  exhibit 
similar  amplitudes  ever  a population  of  epochs  from  the  satca 
classification  yroup.  The  above  pivotal  transformations 
correct  the  variables  in  each  succeeding  matrix  for  linear 
correlation  with  the  selected  variable  so  that  the 
correlated  nature  of  biocyosrnetic  variables  is 
systematically  handled.  However  the  finite  precision  cf 
digital  jj-cocessing  requires  an  audition  il  tolerance  test 
price  to  the  selection  cf  a variable  to  reduce  the 
possibility  or  aegeneracy  when  a variable  is  approximately  a 
linear  combination  of  ether  variaules.  Pr.us  if  tue  magnitude 
or  a uia.jonai  element  cf  the  matrix  in  eg.  3.13  is  less  than 
a tolerance  cf  0.01,  that  variable  is  i.ot  a candidate  for 
selection . 


u 
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4 EXP  EBIHE1TAL  BESOLTS 

This  chapter  presents  a summary  of  the  results  obtained 
in  the  laooratory  with  an  initial  group  of  seven  subjects  cn 
tne  maze  control  experiment.  The  subjects  are  five  females 
and  two  males.  Individual  experimental  sessions  of  about  two 
hours  were  conducted  as  cescribed  below. 

METHODS 

Standaia  Grass  silver  disc  electrodes  were  applied  with 
eiectroconductive  paste  at  six  locations  on  the  scalp  and  to  . -*1 

both  ear-lobes,  ana  electrode  impedance  was  always  lass  than 
10,000  ohms.  A summary  of  the  electrode  sites  and  channel 
configurations  is  presented  in  Figure  4-1.  Channel  eight 
(Fpz  - Oz)  was  used  as  an  artifact  detection  channel.  The 
logic  for  detecting  an  artifact  involves  counting  the 
amplituue  excursions  tint  exceed  a threshold  in  a specified 
time  interval.  The  analog  EEG  signals  are  amplified  over  the 
bandwidth  or  1.0  to  70. 0 Hz.,  and  digitized  every  4.0 
milliseconds.  A data  epoch  consists  of  a set  of  samples 
coilecteu  both  oefoio  anti  after  •>  visual  stimulus. 

Stimuli  consist  ul  brief  (30  micrcsec.)  flashes  or  a 

7 

xonen  st cone  light  (10  lax  illuminance  in  a collimate! 
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I 


FIGURE  4.  I 


Tops  Diagrammatic  representation  (after  Michael  & Halliday,  1971) 
of  the  relative  positions  and  orientations  of  hypothetical  di- 
poles connectea  with  the  central  and  peripheral  parts  of  the 
upper  (at  5 o'clock  and  6 o'clock)  and  lower  (3  o'clock  and  4 
o'clock)  visual  fields.  These  dipoles  are  disposed  to  account  for 
the  early  (30  to  100  milliseconds  to  peak)  positive  voltages  seen 
at  electrode  Oz  (referred  to  ears)  when  the  lower  visual  field  is 
flashed  (UP  command,  see  Figure  7,  Channel  I)  and  the  surface 
negativity  at  Oz  following  upper  visual  field  stimulation  (DOWN 
command,  see  also  Figure  7,  Channel  !). 


BOTTOM*  Topographic  projection  of  left  and  right  hemiretinas  onto 
left  and  right  occipital  lobes.  This  configuration  of  the  brain 
accounts  for  the  observed  large  amplitude  of  the  VHP  to  the  RIGHT 
command  in  the  right  occipital  electrode,  ()2-Oz  (see  Figure  7, 
Channel  4),  and  for  the  large  amplitude  of  the  VEP  following  the 
LEFT  command  in  the  left  occipital  electrode,  ()l-0z  (see  Figure 
7,  Channel  3). 
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oeam)  projected  through  three  red  gelatin  filters,  with  peak 
transmission  wavelengths  cr  620  nanometers,  and  a 
checker  hoard  pattern  cf  alternating  opaque  and  clear  squares 
subter.uing  an  angle  of  approximately  77  milliradians  with 
each  square  subtending  3.5  milliraaians.  A background  light 

-'4 

ot  10  ‘ lamoerts  illuminance  is  also  present  to  allow 
accomodation  to  light.  All  experiments  are  conducted  with 
the  subject  seated  inside  a sound  attenuated, 
electrostatically  and  radio  frequency  shielded  room. 

analysis 

The  analysis  of  the  data  is  performed  on-line  by  the 
930  computer.  The  calculation  procedure  is  implemented  in 
Fortran  II  programming  language  and  operates  on  the  training 
set  data  to  generate  a sequence  of  classification  functions. 
These  linear  functions  are  used  to  classify  each  epoch  of 
the  testing  sets  during  which  the  robot  is  under  control  and 
each  epoch  constitutes  a command  to  the  robot.  The 
performance  of  tue  master  to  robot  comma. .a  channel  is 
evaluated  on  the  basis  of  a confusion  matrix  that  is 
generated  fcr  each  testing  set.  The  rows  of  the  matrix  are 
associated  witr»  each  of  the  possible  commands  to  the  robot 
and  the  coiuans  ace  associated  with  each  of  the  possible 
classif icat icrs.  Tnus  there  is  an  additional  column  in  the 
coniusion  matrix  tor  the  default  classification.  Examples 
of  the  confusion  matrix  are  provided  in  Tholes  4-2  and  4-3. 
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FIGURE  4.2 

SUBJECT  SAD7 f BCI-SDA  CONFUSION  MATRICES 
These  confusion  matrices  show  the  performance  exhibit- 
ed by  subject  SAD 7 on  an  a step  BCi-SDA  analysis. 
Training  set  epochs  I to  280,  testing  set  epochs  281  to 
360.  Four  matrices  are  calculated  using  four  posterior* 
probability  threshold  values. 
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FIGUkc  4.J 

PI tA  Classification  of  SA06,/,d  1240  Epochs, 

'fhese^clasai f ication  functions  resulted  from  a twenty 
step  BnD  P7M  run  or  stepwise  discriminant  analysis  on 
four . experiments  using  subject  dAD.  This  is  a test  or 
stability  over  a three  month  period,  fhe  jacknired 
cl assi ri cation  performance  simulates  a testing  set  of 
new  data. 
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Tue  elements  of  this  matrix  are  generated  oy  recording  the 
history  or  the  classification  process  on  each  epoch  of  a 
given  set.  For  example,  if  the  correct  command  is  that  cf 
the  first  row  and  the  linear  classification  function  selects 
the  one  with  the  second  row,  an  entry  is  added  to  row  one, 
column  two.  The  diagonal  elements  of  the  matrix  contain  the 
correct  classifications  ana  the  non-diagonal  elements  reveal 
the  erronous  classifications.'  Normalizing  the  entries  in 
the  matrix  provides  estimators  of  the  conditional 
probabilities  of  correct  command  interpretation  b/  the  robot 
anu  these,  in  combination  with  a priori  probabilities  for 
the  commands,  form  the  basis  for  measuring  the  performance 
or  tue  master  to  robot  communication  channel. 
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TABLES  1 to  12 


Summaries  of  the  results  produced  online  during  the  experiments  are  shown 
in  Tables  1 through  12.  The  mutual  in > - tion  measure  includes  the  robot  in 
the  smart  mouse  conditions.  The  maximum  is  the  best  40  epoch  performance  seen 
online.  Typically,  nine  recursions  were  performed,  for  a total  of  360  epochs. 
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n. 


TABLE  1 


SUMMARY  OF  EXPERIMENTAL  RESULTS 


BEST  TESTING  PERFORMANCE  (DUMB  MOUSE) 


Threshold  = 0.60  Threshold  = 0.25 

Perfortrar.ee 

Measures  Percent  Percent  Mutual  Percent  Mutual 

Correct  Default  Intcrm.  Correct  Inform. 

Subject  - Date 


SAD7 

- 

Ce2  J 

97.4 

5.0 

1.83 

94.9 

1.81 

SAD  a 

- 

No  1 B 

97.  1 

15.0 

1 .60 

90.0 

1. 60 

JJV2 

- 

Oc  19 

9 4.3 

12.5 

1.62 

90.0 

1.57 

SACO 

- 

Fe  1 9 

94.7 

5.0 

1 .70 

90.0 

1. 51 

SADd 

- 

Le3G 

8 9.0 

8.7 

1.42 

38.7 

1. 46 

EAR  1 

- 

DeO  1 

85.2 

32.5 

1.15 

76.9 

1.01 

MDE2 

- 

Oc  14 

77.2 

1C. 8 

1.00 

74.2 

1. 00 

E AC  1 

- 

Noli 

83.3 

25.0 

1.04 

75.0 

.96 

SAG4 

- 

De22 

75.0 

40.0 

.6  1 

60.5 

. 78 

SAGS 

- 

Mo  2 0 

7 o . 0 

3C.0 

.97 

65.0 

.69 

SDC  1 

_ 

Oc  1 o 

b 5 . 7 

12.5 

.59 

o7.  5 

. 69 

?*» 


I 


TABLE  2 

EXPERIMENTAL  RESULTS 
SUdJECI:  SAE7  DATE:  23DEC75 


Threshold  = 0.60  Threshold  = 0.25 

Per rormance 

Measures  Percent  Percent  Mutual  Percent  Mutual 

Correct  Default  Inform.  Correct  Inform. 


Training  sets 
Smart  mouse 


Avera  ges 

96.1 

1.9 

1.76 

95.7 

1.76 

Std.  Dev. 

1.4 

1 . B 

.10 

1.8 

.10 

Maximum 

9 8.3 

C. 

1.89 

98.3 

1. 89 

imb  mouse 

Averages 

95.5 

1.8 

1 . 69 

94.  3 

1. 64 

Sta.  Dev. 

.17 

.17 

.01 

.06 

.01 

Maximum 

9 5.o 

1.9 

1.69 

94.3 

1.65 

Testing  sets 
Smart  mouse 


A ve  ra  ges 

9 3.1 

6.9 

1.67 

90.  o 

1.61 

Std.  Dev. 

5.  3 

5.5 

.22 

6.  3 

.22 

Maximum 

9 7.4 

'5.0 

1.81 

97.  5 

1.86 

D u m o mouse 


Averages 

93.5 

3 . 8 

1.69 

92.  9 

1.  68 

Stu.  Dev. 

j . 8 

2.  1 

. 21 

4.7 

. 20 

M a x i m u m 

97.4 

5.0 

1.83 

94.  9 

1.81 
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I A b L E 3 

EXPERIMENTAL  BESULTS 
SO'SJECr:  3 A Lb  LATE:  18NOV75 


Thee 

3 hold  = U. 

60 

Threshold 

= 0.25 

Per tormauce 

MtaaUt &S 

Percent 

Percent 

M u t u a 1 

Percen  t 

M utual 

Cor  rec  t 

Default 

Int  crin. 

Correct 

Inform. 

Teaming  sets 
Smart  mouse 

Averages 

9 4.7 

4.5 

1.62 

93. 1 

1. 59 

Std.  Dev. 

.37 

2.3 

.08 

1.6 

. 11 

Maximum 

9 5.  3 

5.0 

1.63 

95.0 

1.76 

Cubic  ucurt 

Averages 

94.0 

5.8 

1.54 

91.2 

1. 47 

Std.  Dev. 

. 17 

.06 

.03 

.35 

. 01 

Maximum 

9 4.^. 

5.7 

1.57 

91.4 

1. 48 

les  t iny  sets 
Smart  mouse 

A •/  e r a j e s 

5 0 . >3 

3.8 

1 . 59 

38 . 3 

1.51 

Std.  Dev. 

o.2 

1.4 

.26 

b . 0 

. 27 

da  < r m U Bl 

97.4 

5.0 

1 .o7 

95.0 

1.74 

Cur,  L mouse 

Aval  a e s 

3a.  j 

10.0 

1.35 

33.  3 

1.  34 

a c u . ue  v . 

1 2.  9 

7.07 

.36 

9 . 3 

. 37 

t‘i  'is.  1 L.  ti  Hi 

5 7.1 

15.0 

1 . o 0 

90.  0 

1.  60 

EXPERIMENTAL  RESULTS 


SUBdEc!:  J J V 2 DATE:  190CT75 


T l)  r e 

shold  = 0 

.60 

Threshold 

= 0.2 

Performance 

Measures 

Percent 

Percent 

Mutual 

Percen  t 

Mutual 

Cor  rec  t 

Default 

Inc crm. 

Correct 

inform 

Training  sots 
So, act  mouse 

Averages 

6 b . 0 

24.9 

1 .GO 

77.0 

. 99 

Std.  Dev. 

2.9 

15. b 

. 18 

7.0 

. 11 

Maximum 

d d . 0 

16.7 

1.14 

81.5 

1.  05 

Dumb  mouse 

Averages 

69.1 

13.2 

1 . 18 

83.5 

1.  11 

Std.  Dev. 

. 14 

1.1 

.01 

.21 

. 00 

M a x i hi  u ai 

89.0 

12.4 

1.19 

63.  6 

1. 11 

Tasting  sots 
5 1.:  a c t mouse 

Avoid  j ri  o 

do. 2 

B .6 

1.39 

84.4 

1.  35 

5 to.  Uov. 

10.4 

J.2 

. 3b 

11.3 

.41 

Maximum 

9 7.0 

7.5 

1.61 

97.5 

1. 87 

i,iimb  mouse 

A ve cages 

9 0.9 

16.3 

1.40 

67.5 

1.  45 

Std.  D v; v . 

4.  B 

a.  3 

.31 

3.  5 

. 18 

Maximum 

9 4.  .1 

1c. 5 

1 . <>2 

90.0 

1.  57 

53 


J. 


If.BLii  5 

£ X p h i<  I i'i  £ N 1 A L RESULTS 


3U  be  £CT  : u A L 9 

Threshold  = 0 

DATE:  19c 
. 80 

Z376 

T hreshold 

= 0.25 

Te  c tor  ilia  iic  e 

Measures 

Percent 

Percent 

Mutual 

Perce  ii  t 

Mutual 

Correct 

default 

Inf  crm. 

Correc  t 

Inform. 

Training  sets 
Smart  mouse 

HVsrtd^dS 

9 3.1 

4.3 

1.69 

93.0 

1.6b 

S t u . Lev. 

J.  1 

2.8 

. 18 

4.0 

.20 

Maximum 

100.0 

0. 

2.00 

100.0 

2.00 

Dumb  mouse 

Averages 

94.  1 

.80 

1.68 

93.3 

1.63 

Stu.  Lev. 

-- 

-- 

-- 

-- 

Maximum 

94.  1 

.80 

1.66 

93.  3 

1.63 

Testia  , sets 
Sir  a r t mouse 

v e t a g e s 

O V . 6 

3.o 

1 . 5 o 

d7 . 8 

1 . oO 

Std.  Lev. 

.]./ 

2 . 3 

. 1 1 

b.  1 

. 19 

ri  » x iir. u :n 

9 2 . J 

) S 
4-  • -J 

1 . i)5 

92. 3 

1. 64 

Duma  t ja-it: 

.'t  V vi  L .i  ^ C ■£» 

*4.7 

3.0 

1 . 70 

90.0 

1.  51 

S *1  ■ J • . V • 

-- 

-- 

-- 

Me xim urn 

*4.7 

5.0 

1 . 70 

90. 0 

1.51 

54 


1 ABLE  & 

EXPERIMENTAL  RESULTS 


SUBJECT 

: SACS 

DATt,:  3 ODE 

iC7  5 

Threshold  = 0 

. 60 

Threshold 

« 0.2  5 

Performance 

Measures 

Percent 

Percent 

Mutual 

Percen  t 

Mutual 

Correc  t 

Default 

Inform. 

Correc  t 

Inform . 

I raining  sets 
Smart  mouse 

Averayes 

9 5.5 

3.8 

1 .71 

93.5 

1.66 

Std.  Dev. 

1.  1 

2.5 

.07 

1.8 

. 11 

Maximum 

95.0 

0. 

1.81 

95.  0 

1.81 

Dunn  mouse 

Averuyes 

9a.  1 

5.9 

1.57 

92.0 

1. 54 

Stu . Lev. 

.99 

3.2 

.05 

.46 

.02 

Ha  xiutum 

94.  7 

5.7 

1.6  1 

92.  5 

1. 56 

Testing  sets 
S mart  in  o u s -2 

A ve  raj-?  s 

O').  1 

z . 5 

1 . 52 

88.  1 

1. 47 

S t a . Lev. 

b.  1 

2.0 

.27 

4.  7 

. 22 

Ha  x mi  a m 

9 7.4 

2.5 

1.88 

95.  0 

1. 76 

u U lil  U lilt)  Hot 

A V C L ei  J 0 d 

b o . o 

5.  - 

1. 42 

85. 4 

1.  40 

i ^ a . _ v . 

2.2 

J.  1 

.07 

i.  1 

. 12 

i xi .i. u .a 

0 0.0 

‘5 . 0 

1 . 4 8 

88.7 

1.  4b 

55 


T A 13  L £ 7 

EXPLuIMtOXAL  RESULTS 


S O 20  C I 

: £ A H 7 

DATS:  01DEC7O 

XI.  re 

shold  = 0 

. oO 

Threshold 

= 0.25 

Performance 

Measures 

Percent 

Percent 

Mutual 

Percon  t 

Mutual 

Correct 

Default 

Inf  or m. 

Correc  t 

Inform. 

Traininj  sets 
Smart  mouse 

Averages 

88.7 

1 1.9 

1.29 

84.2 

1.24 

StG.  Lev. 

3.4 

5.o 

.25 

4.  5 

.24 

Ma x icun 

v 4 . 7 

5.0 

1 .77 

92.  5 

1.65 

Dumb  mouse 

A v e i u j e e 

b d . o 

20.6 

1.10 

80.0 

1 . 00 

S t vi . i e v . 

.26 

3 . 2 

.04 

.57 

. 00 

Maxi  mu  in 

o d . o 

1 U.d 

1.13 

80.  4 

1.01 

i -J  3 L L t i j L±  *-J  t.  S 

3 i.i  it  ii.oub'0 

A v • mi  jar, 

8 3 • 4 

1C. 6 

1.29 

7b.  8 

1.25 

S *.  j . . a v . 

1 J . V 

0 . u 

.38 

11.4 

. 28 

.1  ■ t X . . u.  m 

9 4 . \j 

7 . 5 

i . 6 5 

90.0 

1.58 

L'um*  , ouse 

VtL  i j fc-  2 

7 4 . 1 

Z /.  % 1 

. 9 4 

72.  1 

. 86 

b - j . Lev. 

j . j 

j • ■:} 

. 14 

4.7 

. 14 

M t X i . a. 

o ~j  . r 

? L • > 

1.15 

7h  . J 

1.01 

56 


Perfo  r inane  t 
Measures 


Training  sets 
Smart  mouse 


TA3Li  3 

LXPERIMENTnL  B c S U L C S 
SULJcCT:  MDE2  DATS:  14CCT75 

Threshold  = 0.60  Threshold  = 0.25 


Percent  percent  Mutual  Percent  Mutual 

Correct  Default  Inform.  Correct  Inform. 


Averages 

0 5.0 

5.5 

1.67 

92.4 

1. 58 

Std.  Lev. 

1.4 

2.7 

. 12 

1.7 

. 08 

M ?.  x i u m 

96.7 

6.3 

1.77 

93.3 

1.66 

Dumb  mouse 

Averages 

9 1.9 

3.9 

1.24 

86. 0 

1. 08 

Stu.  Lev. 

5.0 

7.3 

.56 

9.3 

. 5b 

Maximum 

96.7 

4.7 

1.86 

93.3 

1.73 

; sting  sits 
Smart  mouse 

A V G J-  cl  j - ^ 

6 9.  b 

3.  2 

1.51 

39.  1 

1.47 

!d  t vi  , j..  t-)  V • 

1.5 

1.6 

.13 

4.  2 

. 19 

iia  xi.nti.:. 

9 3.  / 

1 . b 

1.69 

93.  3 

1. 69 

D'J  mo  mouse: 

Averages 

6 5 . b 

12. b 

b b 

64.  5 

. 6 9 

StU.  jJV. 

16.1 

2.  1 

.43 

13.3 

. 44 

[•i  i xi.  uin 

7 7.2 

10.9 

1.00 

74. 2 

1.  oo 

1 


..  .....  .......  , 


TABLE  9 


ZXPliLIMENTAL  RESULTS 
SUBJECT:  t A C 1 BATE:  13NOV75 


Thru 

sholu  = 0. 

60 

Threshold 

= 0.2 

Perforaar.ee 

Measures 

Percent 

Percent 

Mutual 

Percen  t 

Mutual 

Cor  rec  t 

Default 

I uf ocm. 

Correct 

Inform 

raininy  sets 
Soar t mouse 

Averages 

69.3 

17.6 

1.24 

82.  b 

1.19 

Std.  Dev. 

0.4 

7.3 

.36 

7.9 

.35 

Maximum 

97.  3 

7.5 

1.62 

95.0 

1.77 

E u Oi  u mouse 

Averages 

62.1 

2 6.2 

.63 

72.  4 

. 83 

Stu.  UcV. 

. c 4 

3.  1 

.Cl 

2.2 

. 06 

Maximum 

61.0 

2o . 0 

.64 

73.9 

. 87 

k;stin.j  sets 
Smart  mouse 

A Veil  j t ~5 

b 0 . o 

13.6 

1 . 17 

74.  4 

1.04 

S t i . Dev. 

o.  4 

6.3 

.17 

3.8 

. lb 

to  a .s  i a.  u a. 

o 0.7 

12.0 

1 . 33 

77.o 

1.18 

L.  aJ  «■  C .'I.  Oil  j o' 

.1 V ■ . I.  i ) i . * j 

o > . 7 

31.3 

.7o 

0 3.  4 

.72 

Stil.  B'V. 

14.  1 

6 . d 

. 37 

16.6 

. 34 

M a / l ii,  .i  .a 

o J . J 

25.  J 

1 .04 

73.0 

. 96 

58 


IABLL  10 

EXPERIMENTAL  RESULTS 
SUDJECi.:  SAG4  DATE:  22DEC75 


T hrud 

hola  = 0 

. oO 

Threshold 

= 0.25 

Performance 

Measures 

Percen t 
Cor  rec  t 

Percent 

Default 

Mutual 
Inf  orm. 

Percen  t 
Correct 

Mutual 

Inform. 

Training  sets 
Smart  mouse 

Averages 

9 2.9 

1C. 9 

1.47 

67.2 

1.35 

Std.  Oev. 

1.5 

4.2 

.15 

3.4 

. 18 

Maximum 

9 4.7 

5.0 

1.72 

92.5 

1 „ 64 

Dumb  mouse 

Averages 

87.3 

17.2 

1.10 

82.  6 

1.  16 

Std.  Ue  v . 

1.  1 

.85 

.06 

1.6 

. 06 

Maximum 

8 8.0 

16.6 

1.14 

83.7 

1.20 

Testing  sets 
3 mart  mouse 

Aver  i ^ = s 

87.9 

12.  j 

1.33 

83.  1 

1. 28 

Stu.  Dev. 

8.3 

4.6 

.20 

3.  2 

. 14 

Maxima  w 

9 7.0 

17.8 

1.5b 

37.0 

1. 4o 

Dumb  nous- 

A ve  rages 

u 0 . 8 

3 1 . J 

.62 

59.0 

. b8 

b t o . JOV . 

7.4 

lx. 4 

.01 

2.  1 

. 14 

i x i u.  u .(i 

8 4.3 

x 7 . 5 

.62 

b0 . 5 

. 78 
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TABLE  II 


EXL’  ErilKLNlAL  RESULTS 


, 

oiJ  Du  z C T 

i o A G 3 

DATE:  20UOV75 

Ih  ce 

shold  - 0 

. 60 

Threshold 

= 0.25 

Pet  f o r uia  nc  e 

Measures 

Percent 

Percen  t 

H u t ua  1 

Pe  rcen  t 

Mutual 

Correct 

Default 

I nr  cm. 

Correct 

I nr orm . 

Irai  r.in  j sett 
Smart  nouse 

A v e l a u e s 

a 6 . 7 

16.4 

1.10 

81.2 

1.10 

S t U . Dev. 

J.  5 

7.2 

.25 

5.3 

. 25 

Ha  xiiaun 

91.9 

-7.5 

1.54 

90.0 

1. 53 

Duran  nouse 

Averages 

d 3.  9 

2 1.2 

. 96 

77.3 

. 95 

itu . Dev. 

.42 

C. 

.02 

. 50 

.01 

M i x i .ti  u ra 

84. 2 

2 1.2 

.97 

73. 2 

. 95 

lestinq  sets 

S i : U L t 1.:  3U&S 

AVt  L i <-  s 

7 7.1 

6.  1 

1.1b 

7 3.6 

1 .0s 

jUl.  Be  v . 

Owl 

b.  J 

.16 

3.  2 

. 09 

M a x i ...  u ,ii 

O 4 t d 

1 /.o 

1.35 

77.0 

1.  17 

D u r.  i a.  o u s c 

::  V r,  t il  J*“S 

7 (;  • d 

30.0 

. 74 

L>  1 . 3 

. b2 

St.i.  lev. 

2 . 

t . 

.04 

5 . 3 

. 1 1 

i i i a i .u  u n 

7 'D  . U 

JO.  9 

. 9 7 
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EXPERIMENTAL  RESULTS 


SUBOECI:  6 DC  1 DATE:  130A775 


Threshold  = O.bO  Threshold  = 0.2 


Per  roc  nance 
Measures 

Percent 

Percent 

Mutual 

Percen  t 

Mutual 

Cor  rect 

Default 

Inf crm. 

Co  erect 

Inform 

Training  sets 
Snar t ncu re 

Averages 

0 4.4 

9.7 

1.57 

91.0 

1.  54 

S t a . Dev. 

2.  3 

2.5 

.21 

3.8 

.21 

Maxi  .r.  u ai 

9 7 > 3 

7.5 

1.89 

97.5 

1.88 

^ u |I>  fc  ’ 0 U i v 

Averages 

8 9.9 

13.7 

1 . 22 

85.0 

1.  20 

6ta.  D-.'V. 

-- 

-- 

-- 

il  a x i n u tti 

8 9.9 

13.7 

1. 22 

85.0 

1 . 20 

Ter  tiny  ^ets 
Sn:art  nous  a 

Averages 

o3.  1 

3.6 

1.48 

84.  4 

1.  45 

S t a . a-:  v . 

7.7 

2.5 

.21 

7.5 

. 1b 

M i X i a.  u i 

9 u . 7 

5.  1 

1.77 

92.5 

1.  b4 

L u in  o c u s 

Aver  i 

b 3 . 7 

1 4 ■ J 

. 5 i 

t>7 . j 

. 69 

She.  Dev. 

-- 

-- 

-- 

U a x l .i  J .2 

t:  D . 7 

14.3 

. 3 9 

b7 . 5 

. 6 9 
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TABLE  13 


Sub  j ec  t .*  SAD7 
CONTRIBUTIONS 


Channel 

Time 

UP 

DOWN 

LEFT 

RIGHT 

OZ-A 

236ms 

43 

8 

22 

7 

PZ-OZ 

136 

100 

6 

39 

45 

02-0Z 

100 

41 

4 

12 

42 

02-0Z 

140 

61 

1 7 

5 

57 

02-0Z 

268 

9 

2 

17 

18 

I-OZ 

132 

74 

30 

34 

1 0 

Table  13: 

Contributions  are  shown  for  each  of  six  variables  chosen  by  a six- 
step  SDA. 


CONTRIBUTIONS 

In  order  to  ascertain  the  contribution  of  each  variable  to  a 
given  group  discriminant  function  (D.F.)  the  roilowing  procedure 
is  adopted,  as  suggested  by  I'atsuoka  (1971);  The  coefficients  of 
the  variables  in  the  Discriminant  ru.nctions  are  a first  approxi- 
mation to  the  degree  of  contribution  each  variable  provides  (on 
the  average)  to  classification  of  the  given  group,  but  these 
values  need  correction  by  a factor  (Vi*)  wnich  can  be  obtained 
in  the  following  manner* 

Vi*  squared  = vi/a.f.,  where  the  Vi  are  the  diagonals  of  the 
residuals  (variances)  in  tne  variance-covariance  matrix,  d.f.  is 
the  number  of  degrees  of  rreedom  associated  with  the  Vi  (d.f.=# 
or  epochs-l).  Each  coefficient  in  the  D.F.  is  then  multiplied  by 
Vi*.  lien,  for  a given  experiment,  these  products  are  normalized 
to  a maximum  of  1 0 J for  easy  comparison.  These  Contributions  are 
tabulated  in  TABLE  13  , 14  , and  lb. 
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TAGLE  15 

AVERAGED  CONTRIBUTIONS 


UP 

DOWN 

LEFT 

RIGHT 

Oz-A  averages 

42 

18 

’8 

27 

Pz-Oz  averages 

74 

44 

45 

42 

01 -Oz  averages 

33 

29 

40 

31 

02-0z  averages 

84 

89 

51 

103 

I-Oz 

48 

16 

20 

11 

TABLE  15;  Average  contributions  are  shown  for  all  subjects,  as  a function 
of  channel,  without  respect  to  time.  The  cross  figures  represent  the 
amplitude  of  the  contributions  of  <;ach  channel  to  the  four  commands. 


t 

l . 
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TABLES  16  to  18 


Shown  in  Tables  16,  17,  and  18  are  the  levels  of  classification 
performance  obtained  on  offline  SDA  runs  with  several  sizes  of  training  sets, 
always  using  the  epochs  immediately  preceeding  the  80  epoch  testing  set. 
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SAD  6 5 Training  Sets 


Number  of  Epochs  Train 

in  Training  Set Threshold  .6 


Test 

.6 
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SAD  7 Cl  ass.  Performance  vs.  Training  Set  Size 


Size  Training  % Test  % 

(Number  of  Epochs) 


40 

100.0% 

82.5% 

80 

95.0 

86.2 

120 

94.2 

95.0 

160 

96.2 

92.5 

200 

96.5 

92.5 

240 

95.8 

92.5 

280 

94.3 

96.2 

TABLE  17 
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SAD  8 Seven  Training  Sets:  Online  Experiment 


Number  of  Epochs 
in  Training  Set 

Train 

Threshold  .6 

.25 

Test 

Threshold  .6 

.25 

Mutual 

Info. 

40 

100.0 

100.0 

81.6 

80.0 

1.1666 

80 

94.5 

90.0 

83. '0 

86.2 

1.3234 

120 

94.7 

91.7 

87.5 

81.3 

1.1869 

200 

93.0 

90.5 

83.6 

78.7 

1.1830 

240 

93.2 

92.1 

86.5 

87.5 

1.4252 

280 

93.4 

92.9 

89.0 

88.7 

1.4594 

TABLE  18 
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iABLH  lv  shows  the  classi r Ication  performance  achieve-!  by  select- 
ing a more  optimal  subset  of  variables  than  is  provided  bv  th» 
wide  general  window  used  for  most  of  the  online  experiments.  In 
order  to  find  a oetter  set  of  variables,  stepwise  discriminant 
analysis  was  perrormed  repeatedly  on  the  data,  faking  a narrow 
time  slice  of  variables  across  channels  per  run.  l'/h°n  all  the 
time  slices  were  analyzed,  a set  of  usually  60  variables  was 
chosen,  and  the  40  variables  from  this  list  which  had  the  highest 
b levels  were  entered  into  a final  SDA  run,  which  gave  the  per- 
formance indicated.  Typical  f. dining  sets  were  200  epochs,  and 
testing  was  performed  on  80  epochs. 


SU3JECT  MODE  Threshold=.6  Thresholds. 25 

r-  /~\  v- M T . • 


% corr 

% der 

M.  I. 

% corr 

M.  I. 

SAD/ 

Train 

96 .5 

1 .0 

1 . 16 

96.0 

1 .75 

Test 

98.  / 

2.4 

1.90 

97.5 

1 .85 

SAD6 

Train 

95.9 

2.0 

1 .69 

95.5 

1 .70 

Test 

94.8 

3.6 

1 .68 

93.8 

1 .66 

JJV2 

Train 

93.2 

1 1 .5 

1 .45 

90.0 

1 .43 

Test 

88.0 

6.1 

1.36 

87.5 

1 .37 

SAD9 

Train 

96.9 

0.6 

1.82 

96.2 

1 .78 

Test 

90.  / 

6.1 

1 .44 

88.7 

1 .42 

SAU3 

Train 

97.9 

4.0 

1 .80 

9d  . 5 

1 .71 

Test 

90.8 

4.9 

1 .53 

87.5 

1 .37 

EAH  1 

Train 

88.9 

10 

1.30 

85.5 

1 . 25 

Test 

72.2 

9.9 

0.82 

70 

.81 

MD82 

Train 

94.3 

1 1 .9 

1 .50 

90.6 

1 .49 

Test 

86  . I 

9.9 

1 .20 

30.0 

1 .06 

EAOI 

Trai  n 

86.8 

24 

.99 

//.  5 

.94 

'lest 

68.4 

28.6 

.50 

61  .2 

.51 

SAG4 

Train 

89.2 

12 

1.22 

84 . o 

1.13 

Test 

73.1 

16. 1 

. 74 

6/. 5 

.72 

SAD  3 

Tr  a i n 

37.3 

21  .6 

1 .01 

80.0 

1 .05 

Test 

68.3 

19.9 

.50 

68.3 

.62 

SDO 1 

Trai  n 

9/.  4 

4.8 

1 . 13 

94.4 

1 .64 

I jst 

9 1 .o 

1 1 .2 

1 .43 

37.5 

1 .39 

A VO 

Train 

93.1 

9.4 

1 . D 

39.6 

1 .5 

S.D. 

4.3 

7.9 

.3 

.3 

lest 

8 3 . 9 

IO.j 

1 .2 

o 0 • V 

1 .2 

S.D. 

1 1 . 1 

1.9 

. 0 

12.1 

„ a 

FIGURES  1 to  6 


These  figures  are  3-D  plots  of  single  epochs  of  visual  evoked  responses 
from  sample  11  (44  msec)  on  the  left  to  sample  70  (280  msec)  on  the  right. 
Voltage  is  on  the  y axis,  and  epoch  number  on  the  z axis. 

The  left  top  and  bottom  plots  are  ar  end  view  (z  axis  perpendicular  to 
the  page),  with  hidden  lines,  of  50  epochs.  The  next  two  plots  are  top  and 
bottom  views.  The  8 samples  selected  by  a 10  step  SDA  run  (on  one  step,  a 
variable  was  removed)  are  marked  as  are  the  first  and  last  samples. 
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FIGURES  / to  I / 

AVERAGES  OF  VISUAL  EVOKED  RESPOMES  FROM  ONLINE 
EXPERIMENTS 

Figures  7 to  I 7 present  averaged  evoked  responses  to  the 
four  command  stimuli.  Each  figure  includes  the  averages 
ror  all  5 channels  of  data  taken  from  one  subject.  The 
baselines  have  been  corrected  by  subtracting  the  average 
of  the  samples  4 through  10  from  each  of  the  samples.  The 
first  3 samples  in  each  epoch  have  been  deleted. 

Full  scale  peak  to  peak  amplitude  is  16  microvolts  for 
those  plots  with  a scale  factor  of  2.6,  and  3 microvolts 
for  those  with  a scale  factor  of  5.0.  The  flash  occurred 
at  t=0. 

The  arrows  mark  the  best  6 samples  found  by  the  stepwise 
discriminant  analysis  program  based  on  a training  set 
consisting  of  the  first  96  epochs  in  the  data  set.  They 
are  numbered  according  to  the  rank  of  their  r levels;  the 
sample  with  the  highest  F level  is  numbered  I. 

..object  SA .17,  Figure  /,  is  marked  with  the  best  6 sam- 
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5.  FACILITIES 

The  project  will  continue  to  be  conducted  under  the 
auspices  of  the  Computer  Science  Department,  University  of  Cali- 
fornia, Los  Angeles.  The  principal  facility  for  this  project  is 
the  computer  system  at  the  Brain  Computer  Interface  Laboratory. 
Laboratory  equipment  includes  three  dedicated  computers  (XDS 
930,  XDS  920  and  I ML AC  PDS-I)  with  complete  peripherals  (card 
readers,  card  punch,  rapid  access  drum,  tape  drives  and  line 
printer . 

Experiment  subjects  are  monitored  from  a specially  designed 
shielded  enclosure  that  contains  various  input  devices  and  out- 
put displays  designed  in  a modular  fashion  for  ease  of  interfac- 
ing with  the  digital  system.  The  experiment  is  conducted  from 
an  adjacent  room  containing  the  control  terminals  to  the  system 
computers,  the  recording  equipment  for  EEG  and  other  biosignals, 
as  well  as  voice  and  video  communication  devices.  The  amplified 
EEG  signals  are  routed  to  a digitizing  station  capable  of  han- 
dling 50  simultaneous  channels  of  analog  input.  During  experi- 
ments, a dedicated  DS  930  computer  with  I6K  words  of  core 
memory  and  2M  characters  on  magnetic  drum  acts  as  data  input 
controller  and  real-time  experiment  controller.  All  real-time 
processing  functions  are  performed  by  the  930  which  also  creates 
complete  experiment  records  for  off-line  batch  processing. 
These  contain,  for  each  data  "epoch,"  the  experiment  parameters 
(sampling  rate,  epoch  lengths,  etc.)  specified  by  the  experi- 
menter a~>  well  as  selected  results  of  on-line  computation,  sub- 
ject responses,  etc.  The  930  also  controls  an  IMLAC  PDS-I  mini- 
computer and  display  terminal  with  8K  of  memory  which  is 
reserved  for  the  generation  of  visual  feedback  displays  for  the 
subject  (.MASTER)  and  information  for  the  experimenter.  In  addi- 
tion, the  PDS-I  as  a stand-alone  computer  can  perform  extensive 
calculations  and  generate  sophisticated  graphics  including  ani- 
mation. 


For  very  iarga  data  processing  programs,  the  main  computing 
power  is  provided  by  the  campus  IBM  350/91  (Campus  Computing 
Network)  which  is  equipped  with  a large  core  memory  of  4M  bytes. 
The  digitized  iata  reaches  the  IBM  360/91  from  the  laboratory  by 
a special  hard  wired,  dedicated  data  line  that  is  used  to  write 
and  read  efficiently  into  and  from  the  360/91  core.  The  data 
transfer  is  controlled  with  a separate  processor  fXDS  920)  which 
has  recently  been  replaced  with  a I6.C  interlaced  version  to 
allow  buffering  and  transfer  without  interference  with  experi- 
ments. A monitor  program  in  the  360/91  controls  both  the  data 
flow  and  the  processing  protocol  from  a privileged  position  with 
respect  to  the  360/91  operating  system  software,  thus  insuring 
imiediate  execution.  Complex  data  analysis  can  be  performed  and 
the  results  fed  back  to  the  laboratory  with  minimum  turnaround 
time.  The  "awakening"  of  this  software  system  anJ  all 
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subsequent  file  handling  are  placed  under  the  campus  timeshared 
system  (URSA)  and  controlled  by  an  IBM  3277  terminal  in  the 
laboratory. 

Finally,  the  BCI  laboratory  computer  system  has  wired-in 
direct  access  to  the  ARPA  Network.  The  network  is  being  used 
for  accessing  and  transmitting  data  to  other  facilities  (e.g., 
UCLA-AfS,  CCM,  MIT-MULTICS,  BBN  and  LBL.)  and  for  communication 
with  other  research  groups. 
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